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ON  A  STOCHASTIC  CONTROL  PROBLEM  WITH  EXIT  CONSTRAINTS 


Martin  Day 


ABSTRACT :  In  this  paper  the  logarithmic  transformation  of  Fleming 

[1]  is  used  to  discuss  a  specific  problem  of  controlled  diffusions. 
The  problem  is  to  minimize  a  certain  quadratic  functional  of  the 
applied  drift  while  satisfying  the  requirement  that  the  place  where 
the  process  exits  a  domain  is  not  in  a  specified  subset  of  its 
boundary.  The  main  result  is  that  the  solution  of  this  problem  is 
given  by  the  logarithm  of  a  related  exit  probability. 
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1.  INTRODUCTION  AND  STATIMI  NT  01  Till  1III0KIM 


In  this  paper  we  are  going  to  use  the  logarithmic  transforma¬ 
tion  discussed  in  |1|  to  solve  a  certain  stochastic  control  problem. 
To  see  our  problem  in  perspective,  it  is  convenient  to  start  witli 
the  basic  result  concerning  the  logarithmic  transformation  that 
Fleming  discussed  in  Section  2  of  11 |. 

Let  A  c  lRd  be  a  bounded  domain  with  C ~  boundary.  The 
functions  o(x)  (d x d  non-singular  matrix  valued)  and  b(x) 

QRd  valued)  are  assumed  to  be  Lipschitz  on  lRd  and,  along  with 
o  (x)  ,  are  bounded.  The  Markov  diffusion  process  4  (t)  is 
defined  by  the  stochastic  differential  equation 

d4(t)  =  b(4(t))dt  +  o(4(t))dw 

where  w  is  a  d-dimensional  Brownian  motion.  1 ^  will  denote  the 
first  exit  time  of  *»(t)  (and  also  of  n(t)  below)  from  an  open 
set  I)  <=lRd.  a(x)  is  defined  to  be  o(x)o(x)',  where  the  ' 
denotes  transpose.  Fleming  showed  that  if  T  6  C~(lRd)  and  we 
de f i ne 


■V  1x1  =  lix  l exP  l  ‘ *  (5  (XA  ) ) )  ]  (.1-1) 


4 


* 


C  G 


2 


is  the  solution  of  the  following  stochastic  control  problem: 

min  E[  L(n(s) ,v(s) )ds  +  ♦(n(tA))J  (1.2) 

0  A 

subject  to 

dn(t)  =  v  (t )  dt  +  o(n(t))du>,  n(0)  =  x  (1.3) 

where  the  minimum  runs  over  all  n(t)  which  solve  such  an 
equation  with  v(t)  bounded  and  progressively  measurable.  More¬ 
over  the  minimum  is  achieved  by  using  the  feedback  control  law, 

v*(x)  =  b (x )  -  a (x) (x) . 

Now  suppose  we  try  4’  =  +°°-X  „  in  the  above,  where  N  c  M. 

N 

The  natural  interpretation  of  (1.1)  is 

g(x)  «  Pxl^ (XA)  €  N],  (1.4) 

I (x)  =  -  log  g(x) . 

Q 

In  (1.2),  4  becomes  an  infinite  penalty  for  n(*A)  €  N  .  We  in¬ 
terpret  this  as  adding  the  constraint  n(TA)  e  N  to  (1.3)  and 
replacing  (1.2)  by 

/a 

min  E [  L ( n(s) , v (s ) ) ds ] .  (1-5) 

0 

Our  goal  is  to  prove  that  I (x)  defined  by  (1.4)  is  still  the 


3 


solution  of  this  constrained  control  problem  and  that  v*(x)  given 

by  the  same  formula  is  an  optimal  feedback  control.  The  result  that 

we  will  prove  is  stated  more  carefully  below,  followed  by  some  remarks 

preparatory  to  the  proof.  The  proof  is  given  in  Section  2.  We 

conclude  with  a  simple  example  in  Section  3. 

The  constraint  n(i^)  £  N  forces  us  to  consider  v(*)'s  which 

are  unbounded;  if  v (•)  is  bounded  in  (1.3)  then  n(t^)  has  positive 

probability  of  being  in  any  specified  open  subset  of  3A.  To 

describe  the  collection  V  of  v(’)'-s  to  be  considered  in  the 

x0 

minimization  in  (1.5),  we  begin  by  requiring  that,  associated 

v  €  V  ,  there  be  an  increasing  family  {  Ci*  )  of  o-algebras  and  an 
x0 

adapted  d-dimensional  Brownian  motion  so  that  v(’)  is 

progressively  measurable  with  respect  to  the  .  Next,  we  require 

that  a  progressively  measurable  process  nv(t)  be  defined  satisfying 

(a)  nV(0)  =  x0,  (1.6a) 

(b)  if  D  is  open,  contains  x(1  and  D  c  A ,  then  (1.6b) 

(the  exit  time  of  n  from  D)  is  finite  a.s., 

(c)  for  t  <  the  following  equation  is  satisfied:  (1.6c) 

dnV(t)  =  v  ( t )  d  t  +  o(nv(t))d<*>. 

Here  is  taken  to  be  1  i in  where  D  is  a  sequence  of  sub- 

domains  for  which  I)n  c  Pn+^  and  UPn  =  A.  (The  notation  Pn  f  A 
will  be  used  to  describe  such  a  sequence.)  This  definition  of 
does  not  depend  on  the  particular  such  sequence  used.  Implicit  in 

I 

L 
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(c)  is  the  assumption  that,  for  any  subdomain  1)  as  in  (b)  , 


|  |  V  (  t  )  |  d  t  < 

J  n 


00  a .  s  , 


This  is  sufficient  for  us  to  apply  Ito's  lemma  for  t  <  see  [4]. 

However,  v(t)  may  behave  badly  as  t  ■*  1 ,  ,  consequently  lim  n  (t) 

may  not  exist.  We  define  the  statement  "hVCV^)  ?  N"  to  mean  that 

there  exists  t  ft.  and  x  ?  N  so  that  lim  n' (t  1  =  x,  i.e.  there 
n  A  n 

exists  a  limit  point  of  n'(t),  as  ttt^,  which  is  not  in  N.  The 
following  admissibility  condition  is  now  the  precise  statement  of  the 
constraint  mentioned  previously: 


Pin  Oa  1  t  N]  =  0, 


V  is  the  collection  of  all  those  v  for  which  the  above 
X0 

requirements,  including  the  admissibility  condition,  are  all 
satisfied. 


Theorem :  Let  N  c  34  be  closed  and  x^  £  A. 

La)  Suppose  that  v  £  V  .  Then 

x0 


E[j  L(n  (s)  ,v(s))ds]  >  Ux0) 


Cl. 7) 


C b )  There  exists  v  €  V  for  which  vftl  =  v  (rf(t)),  where 

x0 


v  *  C.  x )  =  blx)  -  a(x)Vl(x), 
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and  equality  is  achieved  in  (1.7). 

0  Q 

Ke  will  use  N  and  N  to  denote  the  interior  and  compliment 

of  N  computed  relative  to  3A . 

Right  away  we  can  dispense  with  some  trivial  cases.  li 
g ( x )  "  1,  then  I ( x )  e  0  and  the  theorem  is  trivial.  (By  the 
strong  maximum  principle  applied  to  equation  (1.8)  below,  g(x)  is 
either  identically  1  or  strictly  less  than  1.  Likewise  it  is 
either  identically  0  or  strictly  positive.)  li  g(x)  e  0,  let  Nn 
be  a  sequence  of  closed  sets  with  non-empty  interiors  so  that 
DN  =  N.  Because  the  interiors  are  non-empty, 

s„(x)  ■  €  NJ  >  0. 

and  gn(x)  ♦  0.  If  ( t )  satisfies  t he  admissibility  condition 
for  N,  then  it  also  does  for  N  .  The  theorem  in  this  case 
imp  lies  that 

A 

I [  L(h'(s) ,v(s) )dsj  >  -log  g  (xn). 

0  n  i' 

Letting  n  -*  ®  we  sec  that 

A 

F. (  L(nv(s)  ,v(s)  )ds]  =  . 

0 

This  shows  that  it  is  enough  to  prove  the  theorem  in  the  case 
0  <  g(x)  <  1. 


h 


The  conditions  on  A,o  and  I'  imply  that  y  €  ('“(A)  and 
satisfies  the  following  equation  on  A: 


,  \  ;>  •  •  g  ♦  \ 

1  ;  ■  i  r  x .  x  .  ; 

i.i  ii  i 


♦  'he  =0 
.  ’"xi 


(1.8) 


w  i  t  h 


1  on  N 


'  } 


h  on  Nc  . 


iThis  can  he  proved  hy  approximating  hy  t'“  functions 

to  obtain  l'“  approximations  of  y  which  satisfy  (1.8).  Now  the 
Schaudcr  interior  estimate  (3|  Theorem  t> .  2  gives  the  precompactncs 
of  these  approximations  and  their  derivatives  up  to  second  order, 
from  this  it  follows  that  y  £  C “ ( A )  and  satisfies  (1.8).)  This 

implies,  since  y  '  0  in  A,  that  1  €  C~(A)  and  has  houndary 
hehav i o  r 


0  on  N 


♦"'  on  N  . 


(1.0) 


One  checks,  using  (1.8),  that  l  must  obey 


i1  -  i  I  a.  .  I  ♦  II  (x,  VI  (X))  in  A 

‘  i.i  1  •'  x  i  • x  i 


(1.10) 


Here,  as  in  l  1  |  , 


II  ( x  ,  p  ) 


»U)p 


b(x)  -P 


=  min  { v*  p  +  I.  ( x  ,  v ) } 
v 

=  (V*)  •  P  «•  I .  ( x  ,  v * ) 


v»here  v  =  b(x)  -  a(x)p  is  the  unique  v  achieving  the  minimum. 
Rewr i t i ng  ( 1 .  10)  as 


0  = 


1 

T 


l  a  .  .  I 
i.j  l-’  XiXi 


m  in  f  \- 
v 


♦  !  ■  l  x  ,  v )  } 


u 


.111 


we  see  why  one  might  expect  the  theorem  to  be  true;  (1.11)  is  the 
appropriate  dynamic  programming  equation. 


ii.  prooi  oi  mi  mi  oki  m 


Consider  part  fb)  of  the  Theorem  first.  Since  v*  is  locally 
l.ipschit;,  the  equation 

dr*  ft )  =  v*fn*(t))dt  *  o(_n*  ft  Ud-’.  n  (d)  =  \  , 

has  a  unique  solution  tor  t  v  (See  [ )  or  1411.  hollowing 

the  standard  procedure  in  such  matters,  we  can  take  any  subdomain  P 
with  Pc  A  and  applv  Ito's  lemma  in  conjunction  with  (.1.11)  and 
the  fact  that  the  minimum  is  achieved  by  v*  to  see  that 

,VT 

I  c  X  0 )  =  I'(j  1  (n  IS)  ,v*tn*(s)))ds]  ♦  I'lll'Mip  a  n).  f.'.ll 

Now  letting  P  t  A  and  1*  f  +-v .  the  first  term  on  the  right 
approaches 

i 

1:1  1  (n*  fs)  ,v*  f  n*  fs)  1  Ids  ) 

-  0 

bv  the  monotone  convergence  theorem.  l'he  second  term  on  the  right 
in  fJ.l)  is  not  so  simple;  even  if  we  already  knew  that  n 
satisfied  the  admissibility  requirement,  since  1  is  unbounded 
near  portions  of  the  boundary,  it  is  conceivable  that  we  get  a 
positive  limit  for  this  term.  We  can,  however,  draw  the  following 
cone lus ions : 

r  lA 

ifx0)  '  r l  i  fn*fs)  ,v  in  fsiihis]  f-2) 

>  0 

Hxj  >  Kinn*iin at))].  u.->) 


The  second  of  these  implies  that  r*(_t) 
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satisfies  the  admissibility 

condition.  To  prove  this  let  (•  =  { I  (.  x )  <  n }  and  A  ^  be  a  sequence 

of  subdomains  with  A  t  A.  Set  I>  =  U  n  A  ;  it 

m  n,m  n  m 

follows  that  t  =  t,„  a  i  .  Bv  (2.3), 

n ,  m  n  rn 


I  lx0)  >  1  [  I  In  1 1(.  A  A  T) )  I 

n  m 


(2.4) 


Since  T  =  lim  t,  , we  see  that 
m  m 


sup  1 ( n  (s ) )  >  n 


0<  s<  t  aT 
A 


implies  that,  for  all  sufficiently  large  m, 


sup  I ( n  (s ) )  >  n . 

0<s<t.  aT 
A 

m 


This  in  turn,  implies  that  I(n*(tr  a  t.  a  T))  =  n.  Using  this 


in  (2.4)  we  have  that,  for  fixed  n, 


n  m 


n  •  P  [  sup  I(n  (s.l)  >  nj 

0  <  s  <  t .  a  T 
A 


<  lim  n-P[  sup  I(n  (s))  >  n] 
0  <  s  <  t  ^  a  T 
m 


<  lim  E[I(n  (tG  a  a  T))]  <  1  (. x0 ) 
in  -*■  ®  ’n  m 


Thus  , 


10 


p[  sup  i (n* (s) )  =  +°°j  =  o. 

0<s<t . 

A 

But  if  nCt^)  £  N  in  the  sense  described,  then  by  (1.9), 
sup  I  ( n  (s))  =  +o°.  This  proves  the  admissibility  condition. 

«<s<\ 

The  property  that  ip  <  ®  a.s.  for  D  cr  A  follows  from  the 
fact  that  v*  is  bounded  on  D. 

In  light  of  (2.2),  the  proof  of  (b)  will  be  complete  once  we 

show 

A  * 

I (x0)  5  E[  L(n  (s),v  (s))dsj. 

U  h 

But  this  is  precisely  the  conclusion  of  part  (a)  of  the  theorem, 
to  whose  proof  we  now  turn. 

Here  we  start  with  the  solution  of  the  equation 

dnV(t)  =  v(t)dt  +  a  (pV ( t )  )  dw ;  n(0)  =  xQ 

associated  with  a  given  v  £  V  .  The  objective  is  to  prove 

x0 

/a 

E  [  L(nv(s)  ,v(s)  )ds]  >  I  (xn)  . 

0  u 

We  assume  that  the  quantity  on  the  left  is  finite,  for  otherwise 
there  is  nothing  to  prove.  The  usual  Ito  calculation  has  the  same 
difficulty  as  before;  how  do  we  know  that  E[I(hV(tp  a  T))]  -*•  0  as 


1 1 


as  P  t  A  and  1  t 


111  is  diflicultv  can  In'  circumvented  bv 


replacing  the  boundary  values  *<*\  lor  l(xl  with  some  bounded 


N 


.d, 


approximations  T^.  Specifically,  take  a  sequence  i  £  (  (1R  ) 

with  0  ^  y  i  -  0  on  a  neighborhood  of  N  (which 

depends  on  n)  and  T^ix)  f  il  x  £  Nv  .  Ibis  is  possible  since 

N  is  closed.  It  we  1 e t 


e  (\1  -  I  I  exp  (  T(t(i.  )1  1 


1  (  x  1  =  1  o  y  I  g  ( x  )  I  , 

n  ■  1  •  n  1 


then  v.  ix)  4  v',  lx)  and  1  lx)  t  Tlxl  .  It  is  sufficient  therefore 
-n  *  v  n 


o  prove  that  for  each  fixed  n. 


I  I 


rlA 


I  (n  ( s  )  ,  v  ( s ) )  d  s  |  -  1  (x ,1 


dust  as  before,  g()  satisfies  (1.")  and  I  satisfies  (1.11).  Now 

however,  I  ^  €  C(A)  with  1 (  x  )  -  (  x  )  on  AA  .  The  standard 

application  ot  Ito’s  lemma  to  equation  (l.ll)  for  1  ^  (  x  )  implies  that 
for  any  subdomain  1'  with  p  e  A, 


I  (x.)  »  1 

n  0 


1  pA  1 


U hvl s  1  ,v(s))ds  ♦  1  inv(i  A  Till. 


Now,  lett  iny,  P  t  A  and  T  f  ♦»  works.  ^  ( \  1  -  0  on 


n(ip  a  11  is  in  this  neighborhood  as  P  t  A  and  T  f  ♦ 


a  neighborhood  ot  N,  and  the  admissibility  condition  implies  that 

w  i  t  h 

probability  one.  Since  1^1  x)  's  ;1  bounded  time t  ion,  the  dominated 
convergence  theorem  implies  that 


i  n^frV»ni  ii 
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E[In(nV  nD  a  T ) )  ]  -*•  0  as  D  -1  A  and  T  t  +°°. 


The  convergence  of  the  L  term  is  the  same  as  before  and  we  conclude 
that 


1 

n 


(x0) 


E[ 


L(n  (s) , v (s ) ) ds  ] 


Letting  n  t  00  completes  the  proof  of  the  theorem. 


III.  A  SIMPLE  EXAMPLE:  THE  BESSEL  PROCESS 

The  theorem  above  implies  that  whenever  we  can  explicitly 
solve  the  Dirichlet  problem  for  the  generator  of  S(t)  on  A  with 
boundary  values  X^,  we  can  give  an  explicit  solution  for  an 
associated  stochastic  control  problem.  As  an  example,  let  d  =  1 
and  A  =  [ 0 , a ]  (for  any  a  >  0).  If  b(x)  =  0  and  o(x)  =  1 
then  £(t)  is  simply  Brownian  motion.  Let  N  =  {a}.  We  have  then 

g(x)  =  PXK(V  =  3]  =  f  , 

I(X)  =  -log(J),  VI(X)  =  ^  , 

V*(x)  =  0  -  (^)  = 
dn*(t)  =  — —  d£  +  dw. 

n  (t) 

•jig 

Thus  n  (t)  is  the  Bessel  process  associated  with  3-dimensional 

Brownian  motion  6(t);  n*(t)  =  |B(t)|,  see  [4],  In  the  present 
l  2 

case  L(x,v)  =  A-  v  .  According  to  our  theorem,  the  Bessel  process 

*  ,  .  ... 

n  (t)  minimizes 


.  MBMMI :  • 


A 


v(n(s) ) 2dsJ 


among  all  dn  =  v(t)dt  +  dw  for  which  p[n(i^)  =  q]  =o  . 
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